Along with many other forms of human interaction, live musical collaboration has been upended by the COVID-19 pandemic. Widespread quarantining and social distancing essentially suspended performances that require precise timing among multiple players – everything from classical symphonies and choir ensembles to jazz quartets and rock bands. And playing together online through teleconferencing platforms such as Zoom and Skype is not a viable option, due to inevitable audio and video lags.

Screengrab of a JackTrip-powered music performance. (Image credit: Chris Chafe)

Into this breach has stepped a free, open-source software called JackTrip. Developed by Stanford University music professor Chris Chafe and colleagues, JackTrip enables virtually real-time sound streaming over the internet. In this way, JackTrip can often reduce latency – the annoying time delay in data transfer – from one geographical location to another to under a critical threshold of 25 milliseconds. While tiny, that smidgen of time is perceivable by human listeners and is enough to throw a coordinated musical performance out of sync.

Having gotten below that threshold on standard, home-to-home internet connections, JackTrip is now reconnecting a growing number of professional musicians, as well as music teachers with their students.

“It was apparent from March 2020 onward that there was an urgent need to connect musicians who were locked down and unable to play together in the way we love to play together,” said Chafe, professor of music in the School of Humanities and Sciences and director of the Center for Computer Research in Music and Acoustics (CCRMA). Among other instruments, Chafe plays an electric cello. “With JackTrip,” Chafe added, “it’s been great getting to jam again in real-time, just as if we were right in the same room.”

A program whose time has come

After originally graduating from Stanford with a doctorate in music in 1983, Chafe joined the faculty in 1989. He and his colleagues began developing JackTrip in 2000, when they realized that the still-fledgling internet was getting fast enough to support sending audio data, potentially empowering musicians to play together remotely.

Chafe and colleagues built JackTrip on an available, open-source, sound server software called Jack Audio Connection Kit (JACK). At one point, the researchers experimented by connecting musicians at three different locations, or over a “triple” connection. That triple was eventually shortened to “trip” and tacked on to “Jack,” giving the program its quirky moniker. “The name just stuck, kind of like a band name,” said Chafe.

JackTrip technically provides what is known as low-latency, bidirectional, uncompressed audio streaming. The audio data flowing from musician A’s computer to musician B’s is uncompressed, meaning it is not converted into a form that takes up less bandwidth. Avoiding compression reduces latency, because having to decompress a file eats up time, while also ensuring high-fidelity, accurate sound.

Until recently, the software had primarily served the niche purpose of connecting university music groups on different continents. JackTrip also enabled major telematic demonstrations, such as a November 2009 event called ResoNations. The performance linked up an ensemble at United Nations Headquarters in New York City to musicians based at the University of California, San Diego; The Banff Centre for Arts and Creativity in Alberta, Canada; Queen’s University Belfast in Northern Ireland; and Dongguk University in Seoul, South Korea.

Over much of its history, JackTrip has required advanced, high-bandwidth lines like those found on college campuses, commercial entities and governmental institutions. Nowadays, though, and especially after the COVID-19 pandemic struck, the developers of JackTrip have modified the software so that it can work effectively over the basic internet connections between residences. It can also accommodate far larger ensembles than in times past. “We’re now doing JackTrip home-to-home, for ensemble sessions and for lessons,” said Chafe.

JackTrip had to undergo some significant changes to run successfully on less-than-blazing internet connections. Originally, the software connected all endpoints in a given musical session to each other; in other words, the cellist in California would be connected to the flutist in France, and the flutist to the sitar player in India, and the sitar to the cellist, and so on. Now it works in a so-called hub mode, in which a central server – for instance at Stanford – takes in all the audio feeds, mixes them and sends them back out to the end-users. Adding in a middle machine like this does add a bit of latency to the overall data transfer, but not so much as to scramble the musicians’ synchronization.

Numerous coders and developers worldwide have made these recent JackTrip upgrades over the last six months, Chafe said. “Up until the pandemic struck, we were generally making only incremental improvements to JackTrip,” said Chafe. “But now we’ve really been working hard on JackTrip so it can benefit the community.”

A new environment for music-making

The JackTrip software runs on the major operating systems of Windows, Apple and Linux. To facilitate ease of use, the software is also available on tiny, affordable computers called Raspberry Pi4s. Kits like those in use at Stanford – called JackStreamers – are available commercially. The recently formed JackTrip Foundation, a nonprofit co-founded by Chafe, is additionally supplying similar, Pi4-based kits and services for Bay Area choirs and other groups.

Because JackTrip handles audio only, for live performances, rehearsals and lessons, participants often typically augment the collaboration with Zoom, Jitsi, or another videoconferencing application. Video, however, is significantly more data-intensive than audio and thus suffers from more latency. So, during music sessions, JackTrip users learn to listen more carefully and not to rely on their eyes for real-time visual cues from collaborators as they ordinarily would in an in-person setting.

“Your ears get bigger, we like to say,” Chafe says of the performative shift. “That said, there is still some value to having video running at the same time. When I hear my collaborator do something amazing improvisationally over JackTrip and I think ‘wow,’ I can then see on the video feed that the collaborator is smiling. That’s a really nice moment of connection between musicians.”

For online concert-like presentations of multiple people playing together, CCRMA postdoctoral research fellow Constantin Basica meshes the audio feeds with the lagging video feeds, giving the appearance of simultaneous, rhythmic playing. Chafe, Basica, colleagues and guest musicians spread across the United States, Australia, Belgium, Canada, Germany, Ireland and Lithuania have taken to playing weekly Quarantine Sessions since March.

“Telematic performance will never replace playing with people in the same room, and then socializing and having a glass of wine afterward,” said Basica. “But I think JackTrip is a really useful tool that we’ll see more of in the years to come.”